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ABSTRACT 

The integration of many life support systems necessary 
to construct a stable habitat is difficult. The correct 
identification of the appropriate technologies and 
corresponding interfaces is an exhaustive process. Once 
technologies are selected secondary issues such as 
mechanical and electrical interfaces must be addressed. 
The required analytical and testing work must be 
approached in a piecewise fashion to achieve timely 
results. A repeatable process has been developed to 
identify and prioritize system level assessments and 
testing needs. This Assessment Selection Process has 
been defined to assess cross cutting integration issues 
on topics at the system or component levels. 
Assessments are used to identify risks, encourage future 
actions to mitigate risks, or spur further studies. 


INTRODUCTION 

Processes have been developed to accomplish many 
system engineering functions. These processes are 
mainly used for some initial exercise. Engineers 
continue to develop such schemes to document 
progress and to plan for future efforts. Some authors 
have commented about the relative ineffectiveness of 
some system engineering processes, at least when used 
as the sole method of evaluation (Jones, 2005). 

Programmatic system engineering tools have been 
developed to help identify and prioritize systems level 
studies, requiring the attention of the engineering team. 
These studies are used to resolve design intent, identify 
overlooked issues, and plan future budgetary allocations. 

While the goals of such processes are specific to any 
particular industry or discipline, the fundamental nature 
of a selection scheme is to promote uniformity, minimize 
additional investment, and maximize benefit. These 
objectives are independent industry or discipline. 

A generic selection process has been outlined. An 
example of a space habitation implementation developed 


by the AH&I (Architecture, Habitation and Integration) 
team has will be presented with emphasis given to 
Integrated Project Team (IPT) member selection, a 
preprocessing filter, and the selection of an assessment 
prioritization scheme. 

DEVELOPMENT OF SELECTION PROCESS 

A GENERIC SELECTION PROCESS 

The authors propose that any selection process should 
adhere to the following guidelines: 

• The selection process must be documented and 
repeatable. 

• The selection process can be used at any 
system, or project level deemed appropriate to 
investigate. 

• The process must allow for management buy-in 
preventing unsanctioned investment. 

• The selection process must be implemented 
within a relevant time frame. 

• The subprocesses used within the selection 
process must be available to the IPT. 

• The IPT must have the latitude to resource 
preferred selection tools. Some companies have 
a required selection standard, which can be 
used. If no standard is required, the IPT should 
have the latitude to determine what tools are 
defendable for use in selection. 

• The selected assessment results should be 
presented as a prioritized recommendation 
based upon a ranking process. 

The resulting generic framework was developed as a 
proposal to address the aforementioned guidelines. 
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(Figure 1) The individual subprocesses and documents 
will now be shown. 

Initial Recommendations List 


The initial recommendation list is an acknowledgment of 
existing assessment work or assumptions. This 
document should incorporate previous assessment 
recommendations. This list also provides means to 
incorporate topics, as recommendations, from non-IPT 
participants. 



Fig- 1 Generic assessment selection process that satisfies the 
general guidelines 

Assessment Solicitation 


An accepted common manager or coordinator should 
initiate the assessment selection process. The 
assessment solicitation subprocess can include three 
distinct subtasks. 

1 . Development of the IPT 

This step includes the selection and recruitment of 
the IPT members who will perform the Assessment 
Selection Process. Best practices for group selection 
based upon teaming dynamics and consensus 
building should be used. The IPT will contact a 
selected evaluation community that is to provide 
information via questionnaires or interviews. 

2. Determination of the Evaluation Community 


A predetermined evaluation community consists of a 
subset of experts to guide the selection process. The 
scope of the solicitation must be documented and 
understood by all parties involved. Individual 
participation forms or tasking agreements should be 
used to document the scope of the activity. 

3. Evaluation Community Solicitation 

The evaluation community is to be notified that 
programmatic information is to be shared for the 
benefit of the assessment team. Interviews, 
questionnaires or surveys can be used to gather 
programmatic and expert based information. 

Filter Process 


The filter process is intended to identify duplicate, off- 
topic, or incomplete assessment subjects. The identified 
topics may be investigated further by the IPT, or 
abandoned. This step is intended to furnish the 
subsequent prioritization subprocess with a relatively 
small, consistent consideration set. The lists can then be 
shared and documented as the Assessment Short List A 
filter subprocess may be excluded from an abbreviated 
assessment process. 

Assessment Prioritization Tool 


The Prioritization tool is used by the IPT to select 
programmatically beneficial assessments. The specific 
tools available for this process vary in complexity and 
effectiveness. Some assessment strategies are based 
upon executive determination while others are 
consensus based. The scheme selected should require 
the final output to be a quantitative ranking of the 
assessment topics. 

The Analytical Hierarchy Process (AHP) is often used 
within the NASA R&D communities with some success. 
(Saaty 1980) The formality required within the AHP 
method makes it a desirable option, but the 3 R’s 
method has been suggested for this purpose. (Jones 
2005) 

Management Approval 

The Management Approval step is used to share the 
IPT’s findings with the accepted management authority. 
These interactions may result in the acceptance of the 
IPT’s findings and subsequent direction to conduct an 
assessment, trade study, or test. 

Should the management team provide additional 
instruction, or if the scope of the activity has changed, 
the IPT may decide to modify the filter and prioritization 
criteria and reuse the previously collected questionnaire 
data. This additional step has been included as the 
Reevaluate List subprocess in figure 1 . 
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Conduct Assessment 

The IPT may elect to follow a predefined assessment 
study process. Such processes are usually defined within 
a given industry’s culture, or business practices. 

Findings of previous assessment selection loops can be 
redirected as input into the Recommendations List. 

Documents Results 


Assessment results should be documented within a 
configuration management system or an archive. 

HUMAN SUBJECT ASSESSMENT PROCESS 
DEVELOPMENT 

Modification of the Generic Process 


There is significant benefit from implementing early 
system engineering principals and conducting evaluation 
level testing to complex NASA related programs. To 
properly address questions and help develop proven 
requirements in habitat systems, will necessitate the use 
of assessments and trade studies. The AH&I 
(Architecture, Habitation and Integration) team was 
developed to help the JSC and NASA communities to 
initially explore the risks associated with the proper 
development of interior habitat environments. AH&I 
through an integrated team from JSC has identified, 
discussed, and collected a list of habitation issues to 
address via the use of system assessments. 

The generic process discussed earlier has been 
developed further to accommodate the specific needs of 
NASA’s Johnson Space Center (JSC). 

Specific implementation of the generic assessment 
process as related to the NASA JSC AH&I assessment 
selection process are discussed in the following example 
(Figure 2). Detail will be given to the development of the 
IPT member selection, the Filter Subprocess, and an 
implementation of the AHP based assessment 
prioritization scheme. 



Document results: 


technical paper. Sign 
off by stakeholders, 
participants and 
project management. 8 



Fig. 2 AH&I Assessment Selection Process 


Integrated Project Team Selection 

Team selection is a central step in the Element Lead 
Assessment Solicitation subprocess. Individual members 
of the IPT are selected as representatives of participating 
organizations. A single point of contact should be the 
representative of all simiiar subject expertise, except for 
those instances where relevant differences are to be 
considered by the IPT. AH&I prefers to limit the IPT size 
to an odd number (3,5,7) for greater functionality. 

The human system development team frequently 
resourced by the AH&I organization includes 
representatives from: 

• Constellation Program office: 
o Advanced Program Office 

o Advanced Extravehicular Activities (AEVA) 

• Space Life Sciences Directorate, in particular: 

o Habitability and Human Factors Division 

o Human Adaptation and Countermeasures 

Division 

• Crew and Thermal Systems Division, in 
particular: 
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Exploration Life Support (ELS) 

o Life Support & Habitability Systems Branch 

o EVA and Space Suits System Branch 

Participants from each organization represent the 
technical breadth of the project. Participation is based 
upon funding allocation and organizational need. 

Filter Process 

The filter process is included within the expanded 
selection process to refine and reduce the assessment 
topics taken to the AHP prioritization scheme. 

The filtering process developed for this application is 
designed to perform the following: 

® Segregate short or long term studies. Short term 
assessment topics studies are those that can be 
completed in four to six weeks 

® Ensure questionnaire forms are completed and 
are consistently documented 

• Group assessment topics according to common 
subject matter 

• Consolidate duplicate topics or entries 

® Identify at least thirteen assessment topics to be 
scored. If the number of independent 
assessment topics exceeds thirteen, consider 
greater consolidation. The maximum 
assessment candidates chosen should be 
numerous enough to represent the independent 
assessment candidates, while minimized to 
reduce the burden of the prioritization 
subprocess. 

• Preprocess the IPT’s suggestions into a short list 
for preliminary approval and discussion. 

A detailed outline of the AH&I Filter subprocess is 

provided in Figure 3. 



(next step) 


Fig. 3 AH&I Example Assessment Filter Subprocess 


AHP Assessment Selection Tool 

The Analytical Hierarchy Process (AHP) was used to 
determine the most beneficial assessment topics. The 
popularity of AHP method and rigor of implementation 
promotes consistency amongst the comparisons made 
by the selection team. Specific knowledge of AHP is not 
required for participation in the comparisons. 

The AHP allows for a comparison model to incorporate a 
hierarchical framework This flexibility allows complex 
comparisons to be evaluated (Finnie et al, 1999). The 
flexibility comes at the cost of additional pair-wise 
comparisons. 
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The AHP has been formalized as a two phased process, 
where the structure of the objectives or goals are defined 
and the second phase is described as a dialog or 
discussion where options or choices are made that best 
satisfy the objectives. 

The AH&I AHP Selection subprocess is defined as the 
four steps below: 

1 . Identify the pertinent factors or goals that dictate 
the usefulness of each assessment topic relative 
to the IPT's direction. The usefulness of each 
assessment topics will be graded against all 
other topics relative to these goals. The structure 
of these goals may be complex or nested. The 
AHP typically uses a 1-9 based scale for the 
pair-wise comparisons within the AHP. Ishizaka 
et al, 2005 demonstrates the sensitivity of the 
AHP 1-9 based scale relative to a “compromise” 
alternative. 

2. Survey or question the participant group that is 
responsible for the evaluation. Pair-wise 
comparisons are to be performed by a subset of 
the management team. The management team 
was chosen to perform the individual pair-wise 
comparisons, for this particular example. The 
authors strongly suggest the appropriate 
responsible parties be required to perform the 
AHP scoring. This provides a separation of 
responsibility between the scientific/engineering 
community and the management community. 

3. Evaluate the consistency of the pair-wise 
comparisons. A valuable feature of the AHP 
method is a consistency measure, called the 
consistency ratio (CR). The ratio indicates the 
consistency of the pair-wise comparisons. Saaty 
suggests that a CR of more than 0.1 indicates an 
inconsistent comparison. Such inconsistencies 
may then be flagged for resolution with the 
management subgroup. 

4. Evaluate the relative weighting of each goal and 
report. Averages of the individual comparisons 
are taken and the final recommendations are 
reported as a consensus of the scoring 
community. Seldom have we seen a significant 
spread among the top five assessment topics 
chosen by the selection team. 

Figure 4 is a pictorial representation of a process that 
satisfies the steps provided above. 



Fig. 4 AH&I Example AHP Assessment Selection 
Subprocess 

The AH&I IPT has developed questionnaires to perform 
the necessary pair-wise comparisons. Benefit to the 
project is determined as the benefit of continued study 
relating to the following objectives: 

• Stowage Design and Requirements 

• Waste handling and processing 

• Food preparation and storage 

• Commonality of processes and hardware 

• Advanced Environmental Monitoring and Control 

• Dust Implications and Studies 

• Advanced Extra Vehicular Activities 

• Radiation Requirements and Integration 

The corresponding objective hierarch is shown in 
Figure 5. The set of beneficial assessments are 
compared within each objective relative to a set of 
criteria to maximize the benefit to habitation systems. 
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Fig. 5 AH&I AHP Objective Hierarchy 

The benefit of an assessment was determined by a 
group of seven (Each represented by a Roman numeral) 
that represented a management point of view. The 
recommendation of each member is provided in Figure 
6 . 
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0.159 
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0.123 
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0.122 

AEMC 

0.134 


VII 


Rad 0.248 

AEVA 0.221 

AEMC 0.136 

Dust 0.131 


Fig. 6 AH&I AHP Individual Results 

The average results were computed and provided in 
Table one. 


Results 

Average 

Rad. 

0.195 

AEVA 

0.184 

Dust 

0.176 

AEMC 

0.152 

Common 

0.097 

Food 

0.070 


Waste 0.068 

Stowage 0.058 

Table 1 AH&I AHP Average Results 


The IPT suggested the top five issues for continued 
study. The IPT was later instructed to investigate 
subtopics within the four areas identified. Subsequent 
work in commonality and AEVA has been performed by 
AH&I IPT. 

CONCLUSION 

Investment in the development of systems engineering 
practices and tools will continue. These efforts will 
always be deemed beneficial as management relies on 
such activities to monitor key decisions and inspect the 
critical details. Such investments are always intended to 
promote uniformity, minimize additional expenditure, and 
maximize benefit. 

A generic selection tool has been developed that has 
aided the engineering community in the development of 
a repeatable selection methodology. This tool was 
modified and applied to the selection of cross-cutting 
issues for study by the NASA’s life support and 
habitation community. Issues identified for future study, 
by the life support and habitation community, have been 
pursued as studies or tests. 

While the AH&I team is satisfied with the progress made, 
some topics for improvement include minimization of an 
independent set of assessment topics, and the 
appropriate balance of management and technical 
comparison factors. The evaluation of many assessment 
topic yields an impractical number of pair-wise 
comparisons. While the total number of assessment 
topics must be minimized, each of the remaining topics 
must represent a unique assessment option. A stringent 
filter subprocess can be used to force greater variability 
between the selection points and minimize the total 
number of assessment topics. The selection process 
attempted to maximize management’s participation in 
the selection process; after all, management will reject 
results they do not approve of. As a result, management 
was asked to participate in the roles of primary goal 
definers, pair-wise evaluators, and final approvers. The 
development of the secondary goals, and formulation of 
the assessment topics were addressed by the non- 
management IPT members. We believe this division of 
labor achieved a balance between the distinct 
communities. 
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